Staged Training Report ✓ Complete

Run ID: stage10batch1run
Generated: 2026-02-10 12:49:31
Stages Completed: 1
Total Elapsed Time: 02:00:25

Configuration

Config Defaults Changed Since Last Commit

Parameter	Previous	Current
`runs_per_stage`	1	2
`serial_runs`	None	True

All Configuration Parameters (59 parameters)

Parameter	Value
`total_samples`	10000000
`batch_size`	1
`stage_samples_multiplier`	100000000000
`update_interval`	250
`window_size`	100
`num_best_models_to_keep`	1
`sampling_mode`	Loss-weighted
`loss_weight_temperature`	0.5
`loss_weight_refresh_interval`	50
`stop_on_divergence`	True
`divergence_gap`	0.002
`divergence_ratio`	1.3
`divergence_patience`	10
`divergence_min_updates`	5
`val_spike_threshold`	2.0
`val_spike_window`	15
`val_spike_frequency`	0.75
`val_plateau_patience`	100
`val_plateau_min_delta`	0.0001
`custom_lr`	0.0001
`disable_lr_scaling`	True
`custom_warmup`	-1
`lr_min_ratio`	0.001
`resume_warmup_ratio`	0.05
`plateau_factor`	0.8
`plateau_patience`	20
`preserve_optimizer`	False
`preserve_scheduler`	True
`samples_mode`	Train additional samples
`num_random_obs_to_visualize`	2
`selected_frame_offset`	3
`runs_per_stage`	2
`serial_runs`	True
`clean_old_checkpoints`	True
`enable_baseline`	False
`baseline_runs_per_stage`	1
`run_id`	stage10batch1run
`enable_wandb`	True
`wandb_project`	developmental-robot-movement
`lr_sweep.lr_min`	1e-07
`lr_sweep.lr_max`	0.01
`lr_sweep.phase_a_num_candidates`	5
`lr_sweep.phase_a_seeds`	1
`lr_sweep.phase_a_time_budget_min`	3.0
`lr_sweep.phase_a_survivor_count`	2
`lr_sweep.phase_b_seeds`	3
`lr_sweep.phase_b_time_budget_min`	10.0
`lr_sweep.ranking_metric`	median_best_val
`lr_sweep.min_samples_before_timeout`	1000
`lr_sweep.min_evals_before_stop`	5
`lr_sweep.save_sweep_state`	True
`plateau_sweep.enabled`	True
`plateau_sweep.plateau_ema_alpha`	0.9
`plateau_sweep.plateau_improvement_threshold`	0.0005
`plateau_sweep.plateau_patience`	100
`plateau_sweep.cooldown_updates`	5
`plateau_sweep.max_sweeps_per_stage`	2
`plateau_sweep.min_sweep_improvement`	0.0
`stage_time_budget_min`	180

Timing Summary

Stage	Plateau Sweeps	Sweep Time	Training Time	Stage Total
Stage 10	4	00:55:50	00:37:27	01:33:18
TOTAL	4	00:55:50	00:37:27	01:33:18

Plateau Sweep Details

Total Sweeps: 4
Stages with Sweeps: 1 of 1
Total Sweep Time: 00:55:50
Average Sweep Duration: 00:13:57

Stage 10: 4 sweeps

LR Progression: 1.0e-04 → 1.8e-06 → 1.8e-06 → 1.0e-07 → 1.0e-07

Sweep #	Triggered At (samples)	Wall Time	Selected LR	Duration
1	25,000	00:07:24	1.78e-06	00:14:00
2	50,000	00:30:37	1.78e-06	00:14:02
3	75,000	00:52:52	1.00e-07	00:13:52
4	100,000	01:14:04	1.00e-07	00:13:54

Stage Results

Stage	Best Loss	Stop Reason	Samples Trained	Time	Sweeps	LR (Initial→Final)
Stage 10	0.007568	divergence	18,500	01:33:18	4	1.0e-04→1.0e-07

Total Plateau Sweeps: 4

Stop Reason Breakdown

divergence: 1 stages

Loss Across Full Training Run

Multi-Run Statistics

Total Runs: 2
Average Best Loss: 0.007724 ± 0.000156
Best Overall: 0.007568
Worst Overall: 0.007880

Stage 10 (2 runs)

Run	Best Loss	Stop Reason	Samples	Time	Selected
1	0.007568	divergence	18,500	01:33:18	✓
2	0.007880	divergence	17,750	00:26:35
Mean: 0.007724 ± 0.000156		Min: 0.007568 / Max: 0.007880		Range: 0.000311

Best Checkpoint

Name: best_model_auto_session_so101_should_pan_500_stage10_train_304_stage10batch1run_00110250_cont_val_0.007568.pth
Stage: 10
Hybrid Loss (full session): 0.016286

Learning Rate Timeline with Plateau Sweeps

Stage Progression

Stage	Orig Loss	Train Loss	Time	Samples	Stop Reason
10 ⭐	0.016286	0.007568	01:33:18	18500	divergence

Hybrid Loss Over Original Session (per Stage)

Stage 10 (Best) - Hybrid Loss: 0.016286

Sample Counts

Cumulative Across All Stages

Per Stage

Stage 10 (Best) - Total Samples: 18,500

Best Checkpoint Inference

Selected Frame 3

Action 0

Action 1

Action 2

Random Observations

Observation 325

Action 0

Action 1

Action 2

Observation 187

Action 0

Action 1

Action 2

Individual Stage Reports

Stage 10 Run 1 (Staged) (orig loss: 0.016286) ⭐